Deeply Tensor Compressed Transformers for End-to-End Object Detection

DEtection TRansformer (DETR) is a recently proposed method that streamlines the detection pipeline and achieves competitive results against two-stage detectors such as Faster-RCNN. The DETR models get rid of complex anchor generation post-processing procedures thereby making more intuitive. However, numerous redundant parameters in transformers make computation storage intensive, which seriously hinder them to be deployed on resources-constrained devices. In this paper, obtain compact end-to-end framework, we propose deeply compress with low-rank tensor decomposition. basic idea tensor-based compression represent large-scale weight matrix one network layer chain low-order matrices. Furthermore, gated multi-head attention (GMHA) module mitigate accuracy drop tensor-compressed models. GMHA, each head has an independent gate determine passed value. information can suppressed by adopting normalized gates. Lastly, fully compressed models, low-bitwidth quantization technique introduced for further reducing model size. Based methods, achieve significant parameter size reduction while maintaining high performance. We conduct extensive experiments COCO dataset validate effectiveness our (tensorized) experimental show attain 3.7 times full 482 feed forward (FFN) only 0.6 points drop.